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Abstract 

Harrison  [5]  has  introduced  a  model  called  a  Brownian  network,  which  approximates 
a  multiclass  queueing  network  with  dynamic  scheduling  capability.  In  the  present  paper, 
a  particular  Brownian  network  control  problem  is  considered  that  approximates  a  math- 
ematically intractable  scheduling  problem  for  a  two-station  multiclass  queueing  network. 
A  reformulation  of  the  Brownian  network  control  problem  is  solved.  Linear  program- 
ming is  used  to  reduce  the  reformulated  problem  to  a  singular  control  problem  for  a 
one-dimensional  Brownian  motion.  The  objective  of  the  singular  control  problem  is  to 
minimize  the  long-run  expected  average  holding  cost  of  the  controlled  process  incurred  per 
tmit  of  time,  subject  to  constraints  on  the  long-run  expected  average  amount  of  control 
exerted. 
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1.  Introduction.  This  paper  is  part  of  a  research  effort  to  develop  effective  schedul- 
ing rules  for  multiclass  queueing  networks.  Harrison  [5]  has  introduced  a  type  of  crude 
but  relatively  tractable  stochastic  system  model  called  a  Brownian  network.  A  Brownian 
network  approximates  a  multiclass  queueing  network  with  dynamic  scheduling  capability 
if  the  total  load  imposed  on  each  station  in  the  queueing  network  is  approximately  equal 
to  each  station's  capacity.  Hence,  we  can  approximate  a  dynamic  scheduling  problem  for 
a  queueing  network  by  a  dynamic  control  problem  for  a  Brownian  network. 

We  consider  a  control  problem  for  a  Brownian  network  that  approximates  a  particular 
scheduling  problem  for  a  two-station  multiclass  queueing  network.  The  queueing  network 
scheduling  problem,  which  is  described  in  the  next  paragraph,  has  both  input  control  and 
priority  sequencing  decisions,  and  appears  to  be  mathematically  intractable.  In  this  paper, 
the  approximating  Brownian  network  control  problem,  which  appears  in  equations  (1.1)- 
(1.7),  is  reformulated,  and  the  new  formulation  is  solved  exactly.  In  Wein  [10],  the  solution 
found  in  the  present  paper  was  interpreted  in  terms  of  the  original  queueing  system.  This 
interpretation  yielded  an  effective  scheduling  rule  for  the  queueing  network  scheduling 
problem.  A  simulation  study  in  Wein  [10]  showed  that  the  resulting  scheduling  rule  derived 
from  this  analysis  outperforms  conventional  input  control  and  priority  sequencing  rules  that 
appear  in  the  literature. 

The  queueing  network  scheduling  problem  is  motivated  by  a  scheduling  problem  that 
is  encountered  in  many  factories.  Consider  a  queueing  network  with  two  single-server 
stations  and  K  different  customer  classes.   Customers  of  class  k  =  l,...,/\  require  service 
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at  a  specific  station  s{k)  and  their  service  times  axe  independent  and  identically  distributed 
random  x'axiables  with  finite  mean  rrik  and  variance  5|.  Upon  completion  of  service,  a  class 
k  customer  turns  into  a  class  j  customer  with  probability  P^j  and  exits  the  system  with 
probabihty  1  —  Yli^=\  -P<.j<  independent  of  all  previous  history.  We  assume  that  the  K  x  K 
Markovian  switching  matrix  P  =  (Pkj)  has  spectral  radius  less  than  one,  so  that  all 
customers  will  eventually  exit  the  system.  Because  the  number  of  classes  is  allowed  to  be 
arbitrary,  this  routing  structure  is  almost  perfectly  general. 

The  scheduling  problem  incorporates  input  and  sequencing  decisions.  We  assume 
there  is  an  endless  line  of  customers  who  are  waiting  to  gain  entry  into  the  network. 
Each  customer  in  the  line  has  an  exogenously  specified  class  designation.  These  class 
designations  are  such  that,  over  the  long-run,  the  proportion  of  class  k  customers  released 
into  the  system  is  g^,  where  Xlfc=i  Ik  —  1-  The  vector  q  =  (qk)  will  be  referred  to 
as  the  entering  class  mix.  The  input  decisions  are  to  choose  a  non-decreasing  process 
N  —  {N{t),t  >  0},  where  N{t)  is  the  cvunulative  number  of  customers  injected  into  the 
system  up  to  time  t.  Thus  the  input  decisions  essentially  allow  full  discretion  over  the 
timing  of  the  release  of  customers  into  the  system,  but  do  not  allow  for  the  choice  of  which 
class  of  customer  to  inject. 

The  sequencing  decisions  consist  of  choosing,  at  each  point  in  time,  which  class  of  cus- 
tomer to  process  at  each  server  in  the  network.  Preemptive  resimie  scheduling  is  allowed, 
so  that  service  of  a  customer  may  be  interrupted  at  a  particular  station  when  a  higher 
priority  customer  arrives  at  that  station.  Due  to  the  rather  crude  nature  of  the  Brownian 
approximation  that  is  employed  here,  the  assumptions  made  regarding  preemption  do  not 
have  an  effect  on  the  scheduling  policy  that  emerges  from  the  analysis. 

It  is  assimaed  that  a  holding  cost  c^  is  incurred  for  each  iinit  of  time  that  a  class  k 
customer  spends  in  the  queueing  network.  Also,  there  is  a  specified  lower  boimd  A  on  the 
long-run  average  expected  throughput  rate  of  the  queueing  network.  The  throughput  rate 
of  a  queueing  system  is  the  number  of  customer  departures  from  the  system  per  unit  of 
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time.  Our  queueing  network  scheduling  problem  is  to  choose  the  input  and  sequencing 
decisions  so  as  to  minimize  the  long-run  average  expected  holding  costs  incurred  per  unit 
of  time,  subject  to  a  lower  bound  constraint  on  the  long-run  average  expected  throughput 
rate. 

Since  this  queueing  network  scheduling  problem  appears  to  be  mathematically  in- 
tractable, the  best  hope  for  further  progress  appears  to  be  in  the  analysis  of  cruder, 
more  tractable  models,  such  as  the  Brownian  network  model.  Under  conditions  of  baJ- 
anced  heavy  loading,  a  Brownian  network  approximates  a  multiclciss  queueing  network 
with  dynamic  scheduling  capability.  To  state  these  conditions  more  precisely,  we  define 
the  two-vector  p  =  {p,)  be  the  relative  server  utilizations,  or  traffic  intensities,  for  the  two 
stations.  The  values  of  pi  and  p2  can  be  computed  from  the  switching  matrix  P,  the  vector 
m  =  [nik]  of  expected  processing  times,  the  entering  class  mix  q  =  (qk)  and  the  specified 
average  throughput  rate  A,  as  will  be  shown  later  in  the  introduction.  The  balanced  heavy 
loading  conditions  assume  the  existence  of  a  large  integer  n  such  that  0  <  \/n{l  —  p,)  <  1 
for  z  =  1,  2.  As  a  canonical  example,  one  may  think  of  pi  =  p2  =  .9,  in  which  case  n  =  100 
satisfies  this  condition. 

Under  such  conditions,  the  scheduling  problem  described  above  can  be  approximated 
by  a  dynamic  control  problem  for  a  Brownian  network.  For  a  full  development  of  the 
Brownian  network  model,  see  Harrison  [5].  Only  enough  information  will  be  given  here 
to  determine  the  data  of  the  Brownian  network  in  terms  of  the  original  queueing  network 
parameters.  Also,  most  of  Harrison's  notation  will  be  retained  for  ease  of  reference.  The 
Brownian  network  control  problem  is  stated  in  the  next  paragraph,  but  first  we  describe 
the  set-up  that  will  be  adopted.  When  we  say  that  X  is  a  {p.,a^)  Brownian  motion,  it  is 
assumed  there  is  a  given  (i7,  F,  Ft,  A',  P^),  where  (f),  F)  is  a  measurable  space,  X  =  X{u!) 
is  a  measurable  mapping  of  Q,  into  C(R),  which  is  the  space  of  continuous  functions  on 
the  real  line  R,  Ft  =  (7{X{s),  s  <t)  \s  the  filtration  generated  by  X,  and  P^  is  a  family  of 
probability  measures  on  17  such  that  the  process  {X{t),t  >  0}  is  a  Brownian  motion  with 
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drift  /i,  variance  a"^ ,  and  initiEd  state  x.  Let  Ei  be  the  expectation  operator  associated 
with  Pj.  If  }'  =  {}'(/)' ^  >  0}  is  a  process  that  is  Ft -measurable  for  all  <  >  0,  then 
we  say  that  the  process  Y  is  non-anticipating  with  respect  to  the  Brownian  motion  X. 
More  generally,  we  will  say  that  one  process  Y  is  non-amticipating  with  respect  to  another 
process  A'  when  Y  is  adapted  to  the  coarsest  filtration  with  respect  to  which  X  is  adapted. 
Also,  a  stochastic  process  is  said  to  be  RCLL  if  its  sample  paths  are  right  continuous  and 
have  left  limits  with  probability  one. 

It  follows  from  Section  9  of  Harrison  [5]  that  under  the  balanced  heavy  loading  as- 
sumptions described  earlier,  the  queueing  network  scheduling  problem  described  earlier 
can  be  approximated  by  the  following  limiting  control  problem:  choose  a  pair  of  RCLL 
processes  1'  and  6  (A'-dimensional  and  one- dimensional,  respectively)  to 


...  ,.  1   ^ 

minirmze    limsup  —Ei 


f    Y,CkZk{t)dt 
•^0    ;t=i 


(1.1) 


subject  to  the  constraints  :    Y    and  6  axe  non  —  anticipating  with  respect  to  A',  (1.2) 

Z{t)=X{t)  +  RY{t)-qe{t)   foralH>0,  (1.3) 

U{t)  =  AY{t)   for  all  t  >  0,  (1.4) 

U  is  non  —  decreasing  with  U(0)  =  0,  (1-5) 

Z{t)  >   0   for  alH  >  0,  and  (1.6) 

Vim  sup  ^E[U,{T)]    <   7,    fort  =  1,2.  (1.7) 

The  process  Z  represents  the  A' -dimensional  scaled  queue  length  process  and  describes 
the  state  of  the  system.  That  is,  if  Q  is  the  AT-dimensional  queue  length  process  for  the 
original  queueing  system,  then  the  scaled  queue  length  process  is  defined  by 

Z(0  =  ^,  ^>0,  (1.8) 

where  n  is  the  large  integer  specified  in  the  balanced  heavy  loading  conditions.    Notice 
that,  as  in  Harrison  [5],  exactly  the  same  notation  that  is  used  for  the  scaled  processes 


is  used  in  the  approximating  Brownian  control  problem.  We  retain  this  notation  in  order 
to  emphasize  the  queueing  network  interpretation  of  the  Brownian  network  model.  The 
A'-dimensional  process  Y  represents  the  scaled  centered  allocation  process,  and  the  one- 
dimensional  process  0  represents  the  scaled  centered  input  process.  These  two  control 
processes  correspond  to  the  sequencing  and  input  decisions,  respectively,  in  the  queueing 
network  scheduling  problem.  The  scaled  process  F  =  (Y'k)  is  defined  by 

Yk{t)  =  -^ ^^^-^'  t  >  0,  1.9) 

where  Tk{t)  is  the  cumulative  amount  of  time  devoted  to  serving  a  class  k  customer  in 
the  interval  [0,t]  {Tk  is  a  decision  variable  in  the  original  queueing  network  scheduling 
problem),  and  ajt,  which  is  defined  in  equation  (1.17),  represents  the  long-run  fraction  of 
the  server's  active  time  at  station  s{k)  that  must  be  devoted  to  class  k  in  order  to  maintain 
materiad  balance  in  the  network.  The  scaled  process  0  is  defined  by 

Xnt  -  N(nt) 

0{t}  = ■=^ — -,t>0,  (1.10) 

\/n 

where,  as  mentioned  earlier,  A  is  the  specified  average  throughput  rate  and  N{t)  is  the 
cumulative  number  of  customers  released  into  the  network  in  [0,t]. 

The  two-dimensional  process  U  represents  the  scaled  cumuJative  idleness  process  for 
the  two  stations.  Thus  U  is  defined  by 

U(t)  =  :^,  t  >  0,  (1.11) 

where  I{t)  is  the  cumulative  amount  of  idleness  incurred  by  the  server  at  station  i  in 
the  time  interval  [0,^].  (For  brevity's  sake,  processes  such  as  Z,  Y,  Uand  6  will  often  be 
referred  to  without  the  adjective  "scaled".)  The  K  x  K  input-output  matrix  R  =  (Rkj)  is 
defined  by 

Rkj=m-HSjk-P:k),  (1.12) 
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where  Sjk  denotes  the  Dirac  delta  function,  meeining  that  6jk  =  1  li  j  =  k  and  6jk  =  0 
otherwise.  The  2  x  A'  resource  consumption  matrix  A  =  {A,k)  is  defined  by 

\  0,     otherwise.  ^   '     ^ 

The  A'-dimensional  process  A'  is  a  (6,  E)  Brownian  motion,  but  several  definitions  are 
needed  before  we  can  describe  the  A'-dimensional  drift  vector  6  —  (Sk)  and  the  K  x  K 
covariance  matrix  S  =  (Sj/)-  Let  A  =  (A^)  be  defined  by 

X  =  q\,  (1.14) 

so  that  Xk  represents  the  average  number  of  class  k  customers  that  must  depart  from  the 
system  per  unit  of  time  in  order  to  satisfy  the  throughput  rate  constraint. 

Since  P  was  assumed  to  be  transient,  it  follows  that  R  is  non-singular  and  there  exists 
a  unique  non-negative  A'-vector  /?  =  {/3k)  satisfying  the  flow  balance  equations 

A  =  RI3.  (1.15) 

Letting  C{i)  be  the  set  of  all  customer  classes  k  such  that  s{k)  =  z,  we  define  the  two-vector 
of  traffic  intensities  p  =  {p,)  by 

kec(i) 
We  now  define  the  A'-vector  a  =  (ctjt)  by 

ak  =  —   for  all  it  e  C{i).  (1.17) 

Pi 

Then  the  drift  6  and  covariance  S  of  the  Brownian  motion  X  are 

<5=v^(A-i?a)  (1.18) 

and 

K 

^ji  =  ^[(^km^'PkjiSji  -  Pki)  +  akm-'slRjkRik].  (1-19) 

*:  =  ! 


Inequality  (1.7).  which  expresses  the  throughput  rate  constraint  in  terms  of  the  cu- 
mulative server  idleness  process  t7,  is  the  only  relationship  in  the  limiting  control  problem 
that  does  not  appear  in  the  Brownian  network  formulation  of  [5].  The  two-vector  7  =  (7, ) 
in  (1.7)  is  defined  by 

7.  =  v/^(l-p.).  (1.20) 

Inequality  (1.7)  states  that  the  long-run  average  fraction  of  time  that  server  i  is  idle  is  less 
than  or  equal  to  1  —  p,.  In  Wein  [10],  it  is  shown  that  this  inequality  holds  if  and  only  if 
the  long-run  average  throughput  rate  is  greater  than  or  equal  to  A. 

The  remainder  of  this  paper  is  orgajiized  as  follows.  In  Section  2,  the  Brownian  net- 
work control  problem  (1.1)-(1.7)  is  reformulated  so  that  the  state  of  the  system  is  described 
by  a  two-dimensional  workload  process,  rather  than  a  K-dimensional  queue  length  process. 
The  new  formulation  will  be  referred  to  as  the  workload  formulation,  and  it  is  the  solu- 
tion to  the  workload  formulation  that  is  the  goal  of  this  paper.  The  control  processes  in 
the  workload  formulation  are  (Z,  U,  6),  which  are  A'-,  two-  and  one-dimensional  processes, 
respectively.  In  Section  3,  linear  programming  is  used  to  find  the  optimal  process  Z  in 
terms  of  the  process  U. 

In  Section  4,  the  results  of  Section  3  are  employed  to  reduce  the  remainder  of  the 
workload  formulation  to  a  problem  involving  singular  (or  instantaneous)  control  of  a  one- 
dimensional  Brownian  motion.  That  is,  a  {fi,a^)  Brownian  motion  B  is  observed  by  a 
controller  who,  at  any  time,  may  instantaneously  increase  or  decrease  the  value  of  the 
process  B.  Holding  costs  are  incurred  continuously  at  a  rate  h{W{t)),  where  W  is  the 
controlled  process.  No  costs  are  incurred  when  the  controller  either  increases  or  decreases 
the  value  of  the  Brownian  motion  process.  However,  there  are  constraints  on  the  long- 
run  expected  average  amount  of  control  to  be  exerted  in  the  positive  direction  and  in  the 
negative  direction.  The  objective  is  to  minimize  the  long-run  expected  average  holding 
cost  incurred  per  unit  of  time. 

In  Sections  5  through  9,  we  derive  the  solution  to  this  constrciined  singular  control 


problem.  The  solution  is  characterized  by  two  constants  a  and  b,  where  a  <  b,  and  the 
optimal  policy  keeps  the  controlled  process  W  inside  the  interval  [a,  b]  with  minimal  effort. 
The  corresponding  optimal  control  functionals  are  the  local  times  at  these  boundaries.  In 
Section  5,  a  Lagrangian  approach  is  used  to  move  the  two  constraints  into  the  objective 
function.  The  resulting  unconstrained  problem  has  been  analyzed  by  Taksar  [9],  and  his 
results  are  used  in  Section  6  to  develop  sufficient  conditions  for  optimality  of  the  constrained 
singular  control  problem.  In  Section  7,  a  candidate  policy  that  satisfies  the  constraints  is 
derived,  and  in  Section  8.  a  solution  to  the  optimahty  equations  is  fotind,  which  includes  a 
pair  of  Lagrange  multipliers.  The  special  case  where  the  drift  /i  of  the  Brownian  motion  B 
is  zero  is  treated  in  Section  9,  and  the  solution  to  the  workload  formulation  is  summarized 
in  Section  10. 

2.     The  Workload  Formulation 

The  state  of  the  system  in  the  Brownian  network  control  problem  is  described  by  the 
A'-dimensional  queue  length  process  Z,  by  way  of  the  basic  system  relationship  (1.3).  In 
this  section  the  Brownian  network  control  problem  is  reformulated  so  that  the  state  of  the 
system  is  described  by  a  two-dimensional  workload  process.  In  Harrison  [5],  it  is  shown 
that  the  matrix  R  appearing  in  (1.3)  is  invertible.  Let  us  define  the  matrix  M  =  (M,fc)  by 

M  =  AR-\  (2.1) 

This  2  X  A'  matrix  will  be  referred  to  as  the  workload  proHle  matrix.  Define  the  two- 
dimensional  worJcioad  process  W  =  (Wt)  by 

W{t)  =  MZ{t),  i  >  0.  (2.2) 

The  workload  process  describes  the  state  of  the  system  in  the  workload  formulation.  Also, 
define  the  two-dimensional  vector  v  =  (vi)  by 

V  =  Mq.  (2.3) 
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It  was  shown  in  Wein  [10]  that 

p,  =  v,\     for  2  =  1,2,  (2.4) 

where  A  is  a  positive  number  that  is  a  parameter  of  the  original  queueing  network  scheduling 
problem. 

Define  the  two-dimensional  Brownian  motion  B  =  (B,)  by 

B{t)  =  MX{t),  t  >  0,  (2.5) 

so  that  B  has  drift  MS  and  covariance  MY.M^ .     It  was  shown  in  Wein  [10]  that  the 
two-dimensional  drift  vector  MS  =  —7,  where  7  was  defined  in  equation  (1.20). 

Define  the  workload  formulation  of  the  Brownian  network  control  problem  as  choosing 
RCLL  processes  Z,  U  and  6  (A'-,  two-  and  one-dimensional,  respectively)  so  as  to 


minimize    limsup— E 

T— >oo     J- 


/     Y,CkZk{t)dt 
-^0     k=l 


(2.6) 


subject  to  the  constraints  :    U    and  6  are  non  —  anticipating  with  respect  to  B,  (2.7) 

U  is  non  —  decreasing  with  ^(0)  =  0,  (2.8) 

Z{t)  >   0   for  all  t  >  0,  (2.9) 

hm  sup -E[U,{T)]    <   7.    for?  =  1,2,    and  (2.10) 

MZ{t)  =  B{t)  +  U{t)-ve{t)   for  all   t  >  0.  (2.11) 

Let  us  call  a  pair  of  RCLL  processes  (1,^)  a  feasible  policy  for  the  Brownian  network 
control  problem  if  it  satisfies  equations  (1.3)-(1.7)  and  call  a  triple  of  RCLL  processes 
(Z,  U,d)  a  feasible  policy  for  the  workload  formulation  if  it  satisfies  equations  (2.8)-(2.11). 
The  following  proposition  allows  us  to  analyze  the  workload  formulation  of  the  limiting 
control  problem,  rather  than  studying  problem  (1.1)-(1.7)  directly. 

Proposition  2.1.  Every  feasible  policy  (Y,  6)  for  the  Brownian  network  control  prob- 
lem yields  a  corresponding  feasible  policy  {Z,  U,  0)  for  the  workload  formulation  and  every 
feasible  policy  (Z,  U,  6)  yields  a  corresponding  feasible  policy  (K,  9). 
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Proof.  Let  {Y.O)  be  any  feasible  policy  for  the  Brownian  network  control  problem, 
thus  yielding  an  associated  triple  {Z,U,9)  from  equations  (1.3)-(1.4).  Premulti plying  (1.3) 
by  the  workload  profile  matrix  M  and  using  (1.4),  (2.1),  (2.3)  and  (2.5)  gives  equation 
(2.11).  Since  equations  (2.8)-(2.10)  are  satisfied,  the  triple  {Z,U,d)  is  a  feasible  policy. 

Reversing  the  argument,  suppose  that  {Z,U,0)  is  a  feasible  policy  for  the  workload 
formulation.  Then  constraints  (1.5)-(1.7)  are  satisfied  and,  using  equation  (1.3),  one  can 
define  a  corresponding  control  process  1'  by 

Y{t)  =  R-^[Z{t)-X{t)  +  q9{t)],    foralU>0.  (2.12) 

The  associated  queue  length  and  cumulative  idleness  processes  in  (1.3)  and  (1.4)  are  pre- 
cisely the  processes  Z  and  U,  respectively,  that  we  started  with.  Thus  the  pair  {Y,d)  is  a 
feasible  policy.  | 

We  now  address  conditions  (1.2)  and  (2.7)  on  non-anticipativeness.  Since  the  matrix 
R  is  invertible,  if  the  control  process  Y  is  non-anticipating  with  respect  to  the  Brownian 
motion  X  in  the  limiting  control  problem,  then  the  control  process  U  is  non-anticipating 
with  respect  to  the  Brownian  motion  B  in  the  workload  formulation.  However,  notice  that 
in  the  limiting  control  problem,  6  is  non-anticipating  with  respect  to  A",  whereas  in  the 
workload  formulation,  6  is  non-anticipating  with  respect  to  B.  By  (2.5),  it  can  be  seen  that 
more  information  will  be  available  to  the  controller  of  6  in  the  limiting  control  problem 
than  in  the  workload  formulation.  However,  as  will  be  seen  in  Section  4,  the  solution  to 
the  workload  formulation  remains  unchanged  whether  6  is  non-anticipating  with  respect 
to  X  or  with  respect  to  B. 

The  rest  of  this  paper  is  devoted  to  finding  a  solution  to  the  workload  formulation  (2.6)- 
(2.11).  In  Wein  [10],  the  solution  to  the  workload  formulation  is  interpreted  in  terms  of 
the  original  queueing  network  in  order  to  obtain  effective  scheduling  rules  for  the  queueing 
system. 
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3.     Solving  for  Z  in  Terms  of  TJ 

In  this  section  the  optimal  process  Z  in  the  workload  formulation  is  expressed  in 
terms  of  the  control  process  V .  Suppose  we  are  given  a  process  V  that  satisfies  (2.7),  (2.8) 
and  (2.10).  Then  the  optimal  Z  and  d  processes  are  found  by  solving  the  following  linear 
program  at  each  time  t: 

K 

min       Y  CkZk{t)  (3.1) 

A' 

subject  to    ^  MxkZk{t)  +  Vi9{t)  =  Bi{t)  +  Ui{t)  (3.2) 

k=l 
K 

Y  MikZkit)  +  v^eit)  =  B,(t)  +  U2it)  (3.3) 


k=l 


Zk{t)  >   0,    for  k  =  l,...,K.  (3.4) 


At  each  time  t,  this  linear  program  may  have  a  different  set  of  right  hand  side  values. 
Since  it  would  be  easier  to  analyze  a  linear  program  with  a  static  constraint  set,  let  us 
define  the  dual  variables  7ri(<)  and  7r2{t)  and  state  the  duaJ  linear  program  to  be  solved  at 
time  t: 

max        [B,it)  +  lh(t)]7r,{t)  +  [B2(t)  +  U2(t)]7T2it)  (3.5) 

subject  to    A/u7ri(^)  +  M2jt7r2(i)  <    Ck    for  A;  =  1, ...,  A'  (3.6) 

ViTTyit)  +  V2n2it)  =  0.  (3.7) 

Before  analyzing  the  dual  LP  (3.5)-(3.7),  define  the  one- dimensional  workload  imbal- 
ance process  W  by 

W{t)  =  p2W,{t)  -  p,W2{t),  t>0.  (3.8) 

At  each  time  t,  W(t)  measures  how  imbalanced  the  workload  is  between  the  two  stations. 
When  \Vi{t)  and  ^2(0  ^^^  ^'^  the  same  proportions  as  the  long-run  utilizations  pi  and  p2, 
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respectively,  then  \V{t)  =  0.  The  process  W  becomes  positive  (negative,  respectively)  when 
the  workload  in  the  system  becomes  imbalanced  towards  station  1  (station  2,  respectively). 
By  (2.2),  (2.4),  (2.11)  and  (3.8),  the  workload  imbalance  process  can  also  be  expressed  as 

Wit)  =  p2B,{i)  -  piB2{t)  +  P2lh{t)  -  piU2(t),    t  >  0.  (3.9) 

Returning  to  the  dual  LP  (3.5)-(3.7),  use  (3.7)  to  eliminate  TT2{t)  from  the  problem 
and  use  (2.4)  to  substitute  the  utihzation  levels  p  for  the  vector  v  to  obtain  the  one- variable 
LP 

max    — —TTiit)  (3.10) 

subject  to   c'j^^{p2Mik  -  piM2k)T^i{t)  <   p2    for  A:  =  1,  ...,/\.  (3.11) 

The  dual  LP  (3.10)-(3.11)  allows  us  to  express  the  solution  Z{t)  of  LP  (3.1)-(3.4)  in 
terms  of  IV'(/).  Without  loss  of  generality,  assume  that  the  classes  k  =  1, ...,  A'  are  ordered 
so  that 

argmax  c'^\p2Mik  -  pi^ik)  =  1  (3.12) 

k 

and 

argmin  c';;^ip2Mik  -  PiM2k)  =  2.  (3.13) 

k 

By  complementary  slackness,  iiW{t)  >  0,  then  the  solution  Z{t)  of  the  LP  (3.1)-(3.4) 
must  satisfy  Zk{t)  =  0  for  all  classes  k  ^  1.  Similarly,  if  W(t)  <  0,  the  solution  must 
satisfy  Zk{t)  =  0  for  all  classes  k  ^  2.  Notice  that  the  solution  is  Zkit)  —  0  for  k  =  1, ...,  K 
when  W{t)  =  0. 

When  W{t)  >  0,  by  (2.2)  and  (3.12), 

W2{t)^^W,it).  (3.14) 

Mil 
Combining  this  with  the  definition  (3.8)  of  the  workload  imbalance  process  W,  one  obtains 

Ml 

^«(*)=-T7 ^^-T;-^^'(^)     for   z  =  1,2.  (3.15) 

P2Mn  -  Pi^hi 
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Therefore,  when  \V{t)  >  0,  the  solution  Z{t)  to  the  LP  (3.1)-(3.4)  is 


mt) 


■r  I,  -1 

Zjt(0=  •{  ^^^^^"'"^'"'     '       ~~    '  (3.16) 

0,  in-^i. 


Similarly,  when  \V{f)  <  0,  the  solution  is 


^C) 


[  0,  if  jfc  7^  2. 

By  the  balanced  heavy  loading  conditions  on  the  original  queueing  system  stated  in  Wein 
[10],  and  by  conventions  (3.12)-(3.13),  it  follows  that 

P2M11  -piA/21  >  0  (3.18) 

and 

P1M22  -  P2M12  >  0,  (3.19) 

and  therefore  that  the  solution  Z{t)  >  0  for  all  t  >  0.  The  optimal  queue  length  process 
Z  is  constructed  from  (3.16)-(3.17)  for  all  t  >  0.  Notice  that  this  optimal  process  does  not 
depend  on  the  control  process  6  and  depends  on  the  control  process  U  only  through  the 
workload  imbalance  process  W,  as  described  in  equation  (3.9). 

4.     The  Resulting  Control  Problem 

In  this  section  it  is  shown  that  by  substituting  for  the  optimal  queue  length  process 
Z  using  (3.16)-(3.17),  the  workload  formulation  (2.6)-(2.11)  can  be  reduced  to  a  problem 
of  finding  the  optimal  two-dimensional  cumulative  idleness  process  U.  Define  the  one- 
dimensional  Brownian  motion  B  by 

B[t)  =  p2B,{t)  -  p,B2{t),  <>0.  (4.1) 
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Recalling  that  the  two-dimensional  Brownian  motion  B  has  drift  —7,  by  equation  (1.8)  it 
follows  that  B  has  drift  fj.,  where 

/i  =  V"(Pi  -P2),  (4.2) 

and  has  variance  a'^  =  g^MEM^g^  where 

(4.3) 


Q  = 


P2 

L-PiJ 


Notice  that  when  the  system  is  perfectly  baianced  (i.e.,  pi  =  ^2),  then  /i  =  0.  This  driftless 
case  will  be  discussed  separately  in  Section  9;  until  then,  assume  that  /i  is  nonzero.  Let  us 
also  define  the  processes  R  and  L  by 

R{t)  =  p2lh{t),    t>0  (4.4) 

and 

L{t)  =  p,U2{t),    t>0.  (4.5) 

The  letters  R  and  L  are  used  to  denote  cumulative  amounts  of  rightward  and  leftward 
movement,  respectively,  exerted  by  the  controller  on  the  Brownian  motion  B.  The  con- 
straints (2.10)  determine  an  upper  bound  on  the  long-run  average  expected  amounts  of 
rightward  and  leftward  controls  exerted.  Define  a  policy  as  a  pair  of  non-decreasing  RCLL 
processes  {R,  L)  such  that  R  and  L  are  non-anticipating  with  respect  to  B,  Ej-[R{t)]  and 
Ex[L{t)]  are  finite  for  each  t  >  0  and  each  initial  state  x,  and  R{0)  =  L{0)  =  0.  Asso- 
ciate with  the  policy  {R,L)  the  controlled  process  B{t)  +  R{t)  —  L{t)  for  all  i  >  0.  From 
equations  (2.2),  (2.11),  (4.1)  and  (4.4)-(4.5),  it  can  be  seen  that  the  controlled  process  is 
precisely  the  workload  imbalance  process  defined  in  (3.8);  that  is, 

W{t)  =  B{i)  +  R{t)-L{t)   for  all    <  >  0.  (4.6) 

Thus,  in  the  resulting  control  problem,  the  Brownian  motion  B  can  be  controlled  by 
pushing  it  to  the  right  (left,  respectively),  which  increases  (decreases,  respectively)  the 
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workload  imbalance  of  the  system.  Now  define  the  positive  coefficients  hi  and  /?2  by 


hi  = 


C2 


and 


/l2   = 


P1M22   -P2A/12 


Cl 


(4.7) 


(4.8) 


P2M11  -  P1M21 

If  the  optimal  queue  length  process  constructed  from  (3.16)-(3.17)  is  used,  then  one  obtains 

K 


J2ckZ,{t)  =  h{Wit)), 


k=l 


,vhere 


h{x) 


—  hix,     if  X  <  0, 
/z2X,        if  X  >  0. 


(4.9) 


(4.10) 


Thus  for  each  policy  {R,L),  costs  are  incurred  at  a  rate  h{W{t)).    The  resulting  control 
problem  is  to  find  a  policy  {R,L)  to 


minimize    limsup— .E^ 
T—'oo     -^ 


1 


/    h{W{t))dt 
Jo 

P2(l  -Pi)a^ 


subject  to    limsup  — £'2.[/?(T)]  < 

T— 00     T  P\  -  P2 

Umsupi£,li(r)l  <    '""-''^'". 

T^oo     -t  P\  -  P2 


(4.11) 

(4.12) 
(4.13) 


If  a  solution  {R,L)  to  the  constrained  control  problem  (4.11)-(4.13)  can  be  found, 
then  the  optimal  9  process  is,  via  equations  (3.2)  and  (4.4), 

K 


d{t)  =  v-^[Bi{t)  +  ^  -YMikZkit)]   for  all  t  >  0, 


(4.14) 


where  Z{t)  is  defined  in  (3.16)-(3.17).  Notice  that  the  optimal  6  process  would  be  expressed 
by  (4.14)  whether  9  was  non-anticipating  with  respect  to  the  two-dimensional  Brownian 
motion  B  or  with  respect  to  the  A'-dimensional  Brownian  motion  X .  The  rest  of  this 
paper  is  devoted  to  finding  a  solution  {R,L)  to  problem  (4.11)-(4.13). 
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5.     The  Lagrange  Multiplier  Method 


Instead  of  analyzing  the  constrained  control  problem  (4.11)-(4.13)  directly,  a  La- 
grangian  approach  will  be  used.  In  particular,  let  r  and  /  be  the  Lagrange  multipliers 
corresponding  to  constraints  (4.12)  and  (4.13),  respectively.  The  multipliers  r  and  /  are 
interpreted  as  the  per  unit  costs  of  exerting  the  controls  R  and  L,  respectively.  Associate 
with  a  policy  (i?,  L)  the  Lagrangian  cost  function 


k{x)  =  limsup  —Ei 


f    hiWit))dt -\- rR(T)  +  IL{T) 
Jo 


(5.1) 


With  the  aid  of  the  following  theorem,  the  constrained  problem  (4.11)-(4.13)  can  be  solved 
by  making  an  appropriate  choice  of  multipliers  and  then  minimizing  the  Lagrangian  cost 
function.  This  idea  is  not  at  all  new  in  the  context  of  control  theory;  readers  are  referred  to 
Bellman  [1]  and  Everett  [3]  for  early  applications  to  dynamic  programming  problems.  The 
problem  of  finding  a  policy  (R,  L)  that  minimizes  k{x)  will  be  referred  to  as  the  Lagremgian 
problem. 

Theorem  5.1.    Suppose  r  and  I  are  nonnegative  real  numbers  and  suppose  {R* ,  L*) 
is  a  solution  to  the  Lagrangian  problem.  Furthermore,  suppose 


and 


1  rr,*/T^M  P2(l  -  Pl)^ 

hmsup -Er[R  {T}]    =    

r— oo    J-  Pi  -  P2 


1       rrvTM          Pi(l  -P2)m 
hmsup -Er[L  (T)]    =    . 

T— oo     J-  P\  -  P2 


(5.2) 


(5.3) 


Then  (R*,L')  is  a  solution  to  the  constrained  control  problem  (4.11)-(4.13). 

Proof.  Let  W*  be  the  controlled  process  under  the  optimal  policy  (i?*,L*),  so  that 


W*{t)  =  B{t)  +  R*{t)  -  L*{t)   for  all    ^  >  0. 


(5.4) 


Then  for  all  policies  {R,L) 

limsup— Ei 

T— ►oo     J- 


f    h{W* 
Jo 


{t))dt  +  rR*{T)  +  lL*iT) 


< 
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limsup  —E^ 


/     h{W{t))dt  +  rR{T)^lL{T) 
Jo 


(5.5) 


Rearraxiging  terms  yields 


1 


limsup  —Ej: 
r— oo    J- 


h{\V\t))dt 


<   1 


imsup-E^     /     h{W{t))dt 

T-^oo     T  Jq 


+  r\imsup^E,[RiT)-R*iT)] 

T—'oo     ■'■ 

+  llimsup  ^E,[L{T)-L*{T)]. 
r— oo    J- 


(5.6) 


Since  (5.6)  holds  for  all  policies,  it  certainly  holds  for  the  subset  of  policies  such  that 

limsup  iE.[/?(r)]    <    ei^Llfllt  (5.7) 

T— oo     J-  Pi  -  P2 


and 


limsup -Ex [i(T)]    <    . 

r— oo    i  P\  -  p2 


(5.8) 


However,  by  equations  (5.2)-(5.3),  the  terms 


1 


and 


rlimsup-£;x[i?(r)-i?'(r)] 

T— oo     -^ 


/iimsup;^£;,[L(r)-L*(r)] 

T^oo     J- 


(5.9) 


(5.10) 


are  nonpositive  for  this  subset  of  policies.  Hence  the  policy  (i?*,  L*)  is  a  solution  to  (4.11)- 
(4.13).    I 

Notice  that,  at  this  point,  the  Lagrange  multipliers  r  and  /  Eire  not  known.  In  order 
to  invoke  Theorem  5.1,  we  need  to  find  a  pair  of  multipliers  (r,  /)  and  a  solution  (i?,  L)  to 
the  Lagrangian  problem  that  simultaneously  satisfy  (5.2)-(5.3). 

Given  r  and  /,  the  Lagrangian  problem  is  a  problem  of  singular  (or  instantaneous) 
control  of  Brownian  motion.  The  name  stems  from  the  fact  that  the  state  of  the  controlled 
process  can  be  instantaneously  changed  by  the  controller  and,  as  a  result,  the  optimal 
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control  processes  R  ajid  L  are  continuous  but  singular  (i.e.,  the  set  of  time  points  at  which 
R  and  L  increcise  has  meeisure  zero).  Such  problems  have  been  the  subject  of  much  study; 
see,  for  example,  Harrison  and  Taksar  [6],  Karatzas  [7],  Shreve,  Lehoczky  and  Gaver  [8], 
and  Taksar  [9].  In  particular,  Taksar  [9]  has  solved  the  Lagrangian  problem  defined  by  (5.1) 
for  a  convex  holding  cost  fimction  h  that  is  finite  everywhere  or  infinite  on  a  finite  interval. 
Since  the  holding  cost  function  h  defined  by  (4.9)-(4.10)  satisfies  Taksar's  requirement, 
his  results  will  be  used  in  the  next  section,  where  sufficient  conditions  for  optimality  are 
developed  for  the  constrained  problem  (4.11)-(4.13). 

6.     The  Optimality  Equations 

The  goal  of  this  section  is  to  present  sufficient  conditions  for  an  optimal  solution  to 
the  constrcdned  problem  (4.11)-(4.13).  First,  the  sufficient  conditions  for  optimality  will 
be  stated  for  the  Lagrangian  problem  with  arbitrary  nonnegative  ntunbers  r  and  /.  These 
latter  conditions  were  derived  in  Taksar  [9].  The  sufficient  conditions  for  optimality  of 
problem  (4.11)-(4.13)  are  found  by  combining  Taksar's  results  with  Theorem  5.1. 

Let  us  start  by  defining  a  class  of  policies  that  include  the  optimal  policy.  Hcirrison  and 
Taksar  define  a  special  class  of  policies  in  [6]  called  control  limit  policies.  This  type  of  policy 
brings  the  controlled  process  W  within  a  certain  interval  [a,  b]  instantaneously  and  then 
keeps  it  within  this  interval  while  exerting  a  minimum  amount  of  control.  The  controlled 
process  W  under  such  a  policy  is  the  Brownian  motion  B  reflected  at  the  endpoints  of  the 
interval  [a,b].  A  complete  anedysis  of  this  controlled  process  is  contained  in  Harrison  [4], 
where  it  is  referred  to  as  a  regulated  Brownian  motion.  The  control  functionals  R  and  L 
under  a  control  limit  policy  are  the  local  times  of  W  at  the  points  a  and  b,  respectively. 
More  specifically,  the  control  limit  policy  {R,L)  on  the  interval  [a,b]  is  defined  by 

R{t)=    sup  [a  -  B{s)  +  L{s)]^  (6.1) 

0<a<« 
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and 


L{t)  =    sup  [B{s)  +  R{s)  -  b]+.  (6.2) 

0<J<t 


The  existence  and  uniqueness  of  such  functionals  was  estabhshed  in  Harrison  and  Tciksar 

[6]. 

We  now  present  sufficient  conditions  for  optimality  for  the  Lagrangian  problem  (5.1). 
Suppose  that  an  optimal  solution  to  the  Lagrangian  problem  is  fovmd  and  let  g  be  the 
corresponding  value  of  the  Lagrangian  cost  function  k{x).  Thus  g  is  the  minimal  average 
cost  per  unit  time  (independent  of  initial  state)  and  will  be  referred  to  as  the  gain.  As  is 
standard  with  problems  using  average  cost  criterion,  let  V{x)  be  the  cost  incurred  under 
the  optimal  policy  when  the  initial  state  of  the  controlled  process  W  is  x  minus  the  cost 
incurred  under  the  optimal  policy  when  the  initial  state  of  W  is  0.  It  will  be  assumed  that 
V  has  a  first  and  second  derivative  and  V  will  be  referred  to  as  the  potential  function. 

The  following  proposition  was  proved  in  Taksar  [9]. 

Proposition  6.1.    Suppose  {g,\'{x))  is  a  solution  to 

Min  {rV{x)  +  h{x)-g,r  +  V'{x)J-V'{x)}    =   0  (6.3) 

V{0)    =   0,  (6.4) 

and  suppose  there  exists  an  interval  [a,  b]  such  that 

TV(x)  +  h{x)  -g  =  0      for  a  <  X  <  6,  (6.5) 

V'{x)  —  —r    for  X  <  a,  (6-6) 

V'{x)  =  1      ior  x>h.  (6.7) 

Suppose  that  for  some  constants  A'l  and  A^2 

V{x)  <  Ni  +  N2h{x).  (6.8) 

Then  g  is  the  minimal  average  cost  for  the  Lagrangian  problem  and  the  control  Umit  poUcy 
on  the  interval  [a,  b]  is  an  optimal  policy. 
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We  axe  now  in  a  position,  using  Theorem  5.1  and  Proposition  6.1,  to  state  sufficient 
conditions  for  optimality  of  the  constraSned  problem  (4.11)-(4.13). 
Theorem  6.2.    Suppose  {g,  V{x),  r,  /,  a,  6)  satisfy 

Mm  {TV{x)  +  h{x)-g.r  +  V'{x),l-V'{x)}  =    0,                           (6.9) 

V{0)   =   0,  (6.10) 

TV{x)  +  hix)-  g    =    0      for  a  <  X  <  i,  (6.11) 

V'{x)    =    -r    for  X  <  a,  (6.12) 

V'{x)    =    I      for  X  >  6,  (6.13) 

lim sup  I; E^[Rit)]    =    MIH^^    and  (6.14) 

limsnp  ^EAL{t)]    =    Mlz.^,  (6.15) 

r— oo    J-  pi  -  P2 

where  R  and  L  are  the  control  Umit  pohcies  on  the  interval  [a,b].  Suppose  V  G  C"^ 
and  satisEes  condition  (6.8).  Then  the  optima]  poUcy  {R,L)  to  the  constrained  problem 
(4.11)-(4.13)  is  the  control  hmit  poUcy  on  the  interved  [a,  6]. 

The  requirement  that  the  function  V  G  C"^  is  sometimes  called  the  heuristic  princi- 
ple of  smooth  fit  and  is  often  imposed  when  solving  control  problems;  see,  for  example, 
Benes,  Shepp  and  Witsenhausen  [2].  In  the  next  two  sections  a  solution,  which  is  denoted 
(g*,F*(x),r*,/*,a*,6*),  to  equations  (6.9)-(6.15)  will  be  found. 

7.     A  Candidate  Policy 

The  first  step  towards  a  solution  to  equations  (6.9)-(6.15)  is  to  find  candidate  interval 
endpoints  a*  and  b* .  These  values  lead  to  a  candidate  pohcy  {R*,L*)  defined  by 


R*{t)=    sup  [a* -B{s)  +  L'{s)]-^  (7.1) 

0<3<t 
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and 


L'[t)=    sup  [B{s)  +  R'{s)-b'] 

0<s<t 


•1  + 


(7.2) 


In  order  to  find  these  endpoints,  the  following  lemma  concerning  a  regulated  Brownian 
motion  will  be  needed.  See  Chapter  5  of  Harrison  [4]  for  a  derivation  of  this  result.  Let  us 
first  define  u  by 

(7.3) 


2fx 


V  = 

Lemma  7.1.  Suppose  B  is  a  {ii.cr'^)  Browman  motion,  R  and  L  are  as  defined  in 
(6.1)-(6.2),  and  thus  W  =  B  +  R  —  L  is  a  regulated  Brownian  motion  on  the  interval  [a,  b]. 
Then  W  has  a  fruncafed  exponential  steady  state  distribution  with  density  function 

,i/(i-a) 


p{x) 


ue 


giy(6-a)  _  I 


for  a  <  X  <  b. 


Furthermore, 


and 


lira  ^EJR(t)]   =   -nr^, 


lim   ^EMt)]    =    ^  ,,     , 


(7.4) 


(7.5) 


(7.6) 


The  candidate  interval  endpoints  will  be  derived  by  solving  the  following  problem: 
among  the  class  of  control  limit  policies,  find  the  policy  {R,L)  to 


minimize    limsup— £^i 

T — ►oo      -^ 


1 


J" 

Jo 


h{Wii))dt 

subject  to    limsup  — £'i[i?(T)]    =    , 

r-oo    T  Pi  -  P2 

Iimsup-£i[i:(r)]    =    . 

T-oo     1  Pi  -  P2 


(7.7) 

(7.8) 
(7.9) 


That  is,  we  restrict  the  policy  (R,L)  to  take  the  form  defined  in  (6.1)-(6.2),  and  then 
find  the  values  of  a  and  b  that  solve  problem  (7.7)-(7.9).   Notice  that  problem  (7.7)-(7.9) 
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is  just  the  constrained  problem  (4.11)-(4.13)  with  the  inequality  constraints  (4.12)-(4.13) 
replaced  by  conditions  (6.14)-(6.15),  which  Eire  equality  constraints.  The  first  step  in 
solving  problem  (7.7)-(7.9)  is  to  express  the  constraints  directly  in  terms  of  the  interval 
endpoints  a  and  b. 

Lemma  7.2.  A  control  limit  policy  {R,L)  on  the  interval  [a,  6]  satisfies  constraints 
(7.8)-(T.9)  if  and  only  if 

6-a  =  .-Mn(^41^).  (7.10) 

Proof.  The  result  follows  by  equating  the  right  side  of  equations  (7.5)  and  (7.8)  and 
solving  for  (6  —  a).  The  result  can  also  be  proved  by  equating  the  right  side  of  equations 
(7.6)  and  (7.9).    | 

Since  the  cost  function  h  defined  in  (4.9)-(4.10)  is  convex  and  achieves  its  minimimi 
at  zero,  the  optimal  control  limit  policy  for  problem  (7.7)-(7.9)  will  have  endpoints  a  and 
b  that  satisfy  a  <  0  <  6.  By  Lemma  8.2,  the  search  over  values  of  a  and  b  to  solve  problem 
(7.7)-(7.9)  can  be  reduced  to  a  search  over  values  of  a  such  that 


Proposition  7.3.     The  control  limit  policy  with  interval  endpoints 


(7.11) 


-1,         /  (^1     +^2)^2(1    -Pi)  \  .-,^^ 


and 


\hip2(l-  pl)  +  h2Pi{l  -  P2)J 

solves  problem  (7.7)-(7.9). 

Proof.        By  equation  (7.4),   the  objective  function   (7.7)   can  be  expressed   as 
J   h{x)p{x)dx  for  any  control  limit  policy  on  the  interval  [a,  b].  By  Lemma  7.2,  the  optimal 
value  of  the  endpoint  a  can  be  found  by  minimizing  the  function  /(a)  over  the  interval 

22 


(7.11).  where 

/(a)  =   /  h{x)p{x)dx.  (7.14) 

J  a 

Using  (7.10)  to  substitute  for  (6  —  a)  in  (7.4)  and  using  (4.10)  to  substitute  for  h  yield 

/(.)=7"-^;'(/°-/.,.e-'-'& 

(pi   -  P2)       Ja 

/  V  C2('  -Pi)  /  ,  X 

+    /  /i2Te''('-<'>dx).  (7.15) 

Integration  by  parts  yields 

/(a)    =    (t.(/>i-p2))~'('p2(l-pi)(/ii  +  /i2)e-''" 
+  i^a(^i/!>2(l  -  Pi)  +  /'2pi(l  -  P2)) 
-  ^iP2(l  -  Pi)  -  /J2P1(1  -  P2)  +  /i2Pi(l  -  P2)ln  f^'?!"^'!))-        (7.16) 

VP2(1  -  P\)J  J 

Setting  /'(a)  =  0,  we  obtain 

-vp2{l  -  pi){h,  +  h2)e-''''  +  t^(;iiP2(l  -  Pi)  +  /?2Pi(l  -  P2))  =  0.  (7.17) 

Solving  for  the  endpoint  a  gives  equation  (7.12)  and  substituting  this  value  in  equation 
(7.10)  yields  (7.13).  Taking  the  second  derivative  of  /  with  respect  to  a  yields 

f'i    ^         ^-P2(l-Pi)(/^i+^2)e-°  ,„^_- 

/  (a)    =    -, ^ ,  (7-18) 

(Pi  -P2) 

which  is  strictly  positive  by  definitions  (4.2)  and  (7.3),  thus  completing  the  proof.     | 

Thus  a  candidate  interval  [a*,  6*]  has  been  found  such  that  the  candidate  policy 
(i?*,L*)  defined  by  equations  (7.1)-(7.2)  satisfy  conditions  (6.14)-(6.15).  Notice  that,  as 
expected,  a*  =  0  in  the  limiting  case  /?2  =  0,  and  similarly,  &*  =  0  when  /ii  =  0.  A  candi- 
date gain  g*  can  now  be  derived  by  evaluating  the  Lagrangian  cost  funcion  k[x)  defined 
in  (5.1)  under  the  candidate  policy. 
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Corollary  7.4.     Under  the  candidate  policy  (R*,L*),  the  Lagrangian  cost  function 
k{x)  is 

V  \hiP2{^- pi)  + h2pi{l- p2)J 

{hi  +  /J2)pi(l  -  P2) 


+  h2Piil  -  P2)ln 


hip2{l  -  pi)  +  /j2Pi(l  -  P2) 


(Pl    -P2)  (Pl    -P2) 


Proof.  By  (7.5)-(7.6)  and  (7.14),  the  value  of  the  Lagrangian  cost  function  (5.1) 
under  the  candidate  policy  {R*,L*)  is 

3'  =  •^(«*)  +  %.(6.-t).i  +  \_J,,..a-y  (7-20) 

Substituting  for  {b*  —  a*)  and  a*  using  equations  (7.10)  and  (7.12),  respectively,  yields 
expression  (7.19).     | 

8.     Solution  to  the  Optimality  Equations 

In  this  section  a  solution  (g* ,V*{x),r* ,l*,a* ,b*)  to  equations  (6.9)-(6.15)  will  be 
presented.  In  the  previous  section  candidate  endpoints  a*  and  b*  were  derived  and  a 
candidate  gain  g*  was  found  in  terms  of  the  Lagrange  multipliers  r  and  /. 

We  start  this  section  by  finding  a  pair  of  candidate  multipliers  (r*,/*).  By  condi- 
tion (6.12)  and  the  assimiption  that  V  G  C^,  we  set  V"{a*)  =  0  in  equation  (6.11)  to 
obtain,  upon  setting  V'{a*)  =  —r  and  using  (4.10)  and  (7.12)  to  substitute  for  h  and  a* 
respectively, 

-ll„   f  (^i  +  ^2)/32(l  -  Pi) 


g^-nr-  h,v-^  In     ,       \\'       ^^^^^        ''       ,      •  (8.1) 

\hip2{\  -  pi)  +  h2pi{l  -  P2) 

Similarly,  setting  V'{b*)  =  /  and  V"{h*)  =  0  in  equation  (6.1)  yields 

-1,^  /         (^1  +  ^^2)^1(1  -  P2. 


g  =  pl  +  h2U-'  In     ,       ;/         Vr^       [;' .  (8.2) 

\hip2[\  -  pi)  +  h2p\{l  -  P2). 
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Equating  the  right  sides  of  equations  (S.l)  and  (S.2)  gives 

,   ,           h,        (         {h^+h2)p2{\-pi) 
r  +  I  = In  ' 


fiu       \hip2{l  -  pi)  +  h2Pi{l  -  P2) 


flU        \hip2{l  -  pl)  +  h2pi{l  -  p2)j  ' 

Lemma  8.1.    Lef  r  +  /  be  a^  in  (8.3).  Then  r  +  1  >  0. 

Proof.    By  rearranging  terms  in  (8.3),  we  see  that  r  +  /  >  0  if  and  only  if 

(hr  +  h2)p2il  -  P^)  ^-'■    ^      f  jh,  +  h2)p,{l  -  P2)  V'  ._.. 


.hiP2{l  -  pi)  +  h2pl{l  -  P2)J  \h1p2il  -  pi)  +  h2pi{l  -  P2) 

or,  equivalently, 

— >    {p2{l  -pl))''>+''2  (/9i(l  -^2))"'+"=  ■  (8.5) 

hi  +  h2 

Since  ln(x)  is  an  increasing  concave  function,  it  follows  that  r  +  /  >  0.     | 

It  turns  out  that  there  is  not  a  unique  pair  of  multipliers  (r*,/*)  that  satisfy  the 
optimality  equations  (6.9)-(6.15);  any  nonnegative  pair  that  satisfy  equation  (8.3)  will 
suffice.  To  this  end,  choose  any  A*  G  [0, 1].  The  pair  of  candidate  multipliers  (r*,/*)  are 
defined  by 

A*(_^ln'^         (^1  +^2)p2(l  -  Pi) 


pu       \h1p2il  -  pi)  + h2Pi{l  -  P2) 


and 


^2  ^^  f         (hi  +h2)pi{l  -  P2)  II  .gg. 

pu       UiP2(l-pi)  +  /i2Pi(l-P2), 


r  =  (l-A*)(-^ln^         (^i+/^2)P2(l-Pi) 


pu       \hiP2{\  -  pi)  +  h2pi{\- P2) 

^Mnf         (^1+^2)P1(1-P2)         \\  .g^. 

pu       \hip2{\  -  pi)  +  h2P\{l  -  P2)  )  )' 

With  the  candidate  multipliers  (r*,/*)  in  hand,  it  is  now  possible  to  calculate  the 
candidate  gain  g*  directly  in  terms  of  the  problem  data.  This  can  be  done  in  three  ways: 
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substituting  r*  for  r  in  equation  (8.1),  substituting  /*  for  /  in  equation  (8.2),  or  substituting 
both  r*  and  /*  in  equation  (7.19)  of  Corollary  7.4.  Readers  may  verify  that  all  three  ways 
lead  to  the  result  that 


g*  =(A*-l)/ii//-Mn 


-ii„  I  (^1  +^2)^2(1  -  Pi) 


hiP2{l  -  pi)  +  /l2Pl(l  -  P2) 


,.L       -1,_   /  (^1  +^2)/>l(l  -  P2) 


+  X*h2U-^  In     ,       )/  7^       .7' r     .  (8.S) 

To  complete  the  candidate  solution  of  the  optimality  equations,  a  candidate  potential 
function  V*  is  required.  With  g*,a*  and  6*  now  chosen,  condition  (6.1)  becomes 

^iV'{x)  +  ^a^V"(x)  +  h{x)  -g*  =  0   for  a*  <  x  <  b* .  (8.9) 

The  solution  to  this  second  order  differential  equation  is  given  in  the  following  proposition. 
Proposition  8.2.     The  solution  V*'  to  (8.9)  is 


V*'{x)  =  —  +  C*t-"^  -  ^e-"^   /    h{y)e''^dy     for  a    <x<  h\ 


(8.10) 


where 


^.  _h,fi-'iy-'{h,+h2)p2{l-Pi).^^f'         jh,  +  h2)p2{l  -  Pi)  . 

hxp2il  -  pi)  +  h2pi{l  -  P2)         \hip2{l- pi)  + h2pi{l  -  P2)  '  ' 


Proof.    The  solution  V*'  to  (8.9)  is 

V*'{x)  =  —  +  Ce-"'  -  ^e-"""   f    h{y)e''Uy     for  a'  <  x  <  b* ,  (8.12) 

P  (^^  Ja- 

where  C  is  axi  unknown  constant.   To  find  C,  we  evaluate  (8.12)  at  the  point  x  =  a*,  set 
V*'{a*)  =  —r*  and  substitute  for  a*  using  (7.12)  to  get 

fi  V         [hi  +  h2)p2{l  -  Pi)         J 


or,  equivalently, 


^,  ^ip2(l  -Pi)  +  /i2pi(l  -  P2)  1  ,^... 

g      =    -/xr    -//C     .,      ,    ,    ,      ,. -\ •  (8.14) 

[hi  +  h2)p2[l  -  Pi) 
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Substituting  g*  and  /•*  for  g  and  r  in  (S.l)  using  (8.6)  and  (8.8)  respectively,  equating  the 
right  sides  of  equations  (8.1)  and  (8.14),  and  solving  for  C  yields  the  candidate  constant 
C*  defined  in  (8.11).    | 

The  following  three  lemmas  verify  that  the  boundary  conditions  V*'{b*)  =  /*,  V*"{a*) 
=  0  8Lnd  V*"{b*)  =  0,  respectively,  are  satisfied. 

Lemma  8.3.     The  function  V*'  de£ned  in  (8.10)-(8.11)  satisEes  V*'{b*)  =  I*. 

Proof.    Evaluating  V'*'  in  (8.10)  at  x  =  fc*  and  integrating  by  parts  yield 

V"(b*)  =^—  +  C'e-"^'  -  4-^""**  i-hiu-^i-l  -  {e^^'iva"  -  1))) 

+  h2U-\e''^\vb* -l)  +  \)\.  (8.15) 

Substituting  for  a*,  6*  and  C*  using  (7.12)-(7.13)  and  (8.11),  we  obtain 

r*'(6M  =  ^  +  /         (^1  +^2)p2(l -Pi)         )  ^1  ln(         (^1 +  ^2)p2(l -Pi)         X 
jj.  hip2{\  -  pi)  +  h2p\{\- p2)    p.u        hip2{l  -  pi)  + h2pi{l  -  p2) 

/^lP2(l   -  Pl)  +  ^2pl(l  -  P2).  2/li      /liP2(l  -Pl)  +  ^2Pl(l  -  P2)^ 

^  (h,+h2)p,{l-p2)  '  ^2    ^  ih,+h2)pAl-P2)  ^ 

_^-2  _  ^-2/  (^1   +  ^2)p2(l  -  Pl)  X         .  {hi  +  h2)p2{l  -  p\)  ^ 

hipiil  -  Pl)  +  h2Pi{l  -  P2)  hip2{l  -  pl)  +  h2Pl{l  -  P2) 

-21  {h\   +  h2)p2{l  -  P\)  ^^  2/l2  //llp2(l  -  Pl)  +  ^2Pl(l  -P2)- 


^/?lP2(l-Pl)  +  /i2Pl(l-p2)V  ^'    ^  (/^1+^2)P1(1-P2)  ' 


V 


-2     ,     .,-2 


(^1   +  ^2)Pl(l  -P2)  M     /  (^1   +^2)Pl(l  -P2) 


+'--'(.■'    ':'  ,,7'  .JM 


/llp2(l  -Pl)  +  /i2pl(l  -P2)  /ilP2(l  -Pl)  +  /i2Pl(l  -  P2) 


-21  (^1  +  ^2)pl(l  -P2) 


""        ^/llP2(l-pi)  +  /i2Pl(l-p2)V'  ■ 

Cancelling  and  rearranging  terms  yield 

g*       ^2  ,     (        (hi  +  h2)pi{l  -  P2)         \       ,hi+h2 


//         /ii/        \/jip2(l  -pi)  +  /i2Pl(l  -P2)/  Pl^ 

^1P2(1  -Pl)  +  /^2Pl(l  -P2)\         /ti/)2(l-pi)  /i2  .g  ^-,. 


(/Jl  +/J2)pi(l  -P2)  y         pi/pi(l-p2)         PI^ 

Since  the  last  three  terms  on  the  right  side  of  (8.17)  sum  to  zero,  using  (8.8)  we  see  that 

v*'{b*)  =  r.   I 
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Lemma  8.4.    V'*'  satisfies  V*"{a')  =  0. 
Proof.    Differentiating  (S.IO)  yields 

V"'{x)  =  -uC'e-"'  -  ^^  +  -^  /    h{y)e''ydy   for  a'  <  x  <  b\  (8.18) 

Using  (4.10)  and  (7.12)  emd  evaluating  equation  (8.18)  at  x  =  a*  give 

'      ^°^  =  -^^^         ih,+h2)p2il-p.)         ^ 

^  (-hi       [         {h^  +  h,)p2{l  -  p,)         \ 
a^^  ^u        \h,p2{l-p,)  +  h2P,[\-p2))'  ^         ' 

which  equals  zero  by  (8.11).     | 

Lemma  8.5.    V*  satisfies  V*"{h*)  =  0. 

Proof.    Evaluating  (8.18)  at  x  =  6*  and  integrating  by  parts  yield 


a' 


(-1  -  (e"''  {va'  -  1)))  +  h2V-'{e'"'  (ub*  -  1)  +  1)1.  (8.20) 

Substituting  for  a*,  6*  and  C*  using  (7.12)-(7.13)  and  (8.11),  respectively,  it  follows  that 

V"'(b*)  =      v(         (^1 +  ^2)/>2(l -Pi)         )^^  \n(         (^1  +  ^2)^2(1  -  Pi)         ^ 
hip2{l  -  Pi)  +  h2pi{l  -  P2)    ^11^        hiP2{l  -  pi)  +  h2pi{l  -  P2) 

,hiP2{l-  pl)  +  h2pl(\  -  p2)s        h2       .  (hi  +  h2)piil  -  P2)  X 

{hi  +  h2)pin-  P2)  fi    ^^hip2il-pi)  +  h2Pi{l- P2)^ 

2     -uhiP2il  -  pi)  +  h2Plil  -  P2).f,        -2     ,     L        -2 


(/)l   +/Z2)Pl(l  -P2)  V 

/         {hi  +  h2)p2{^  -  p\)         X  .    /         {hi  +  h2)p2{l  -pi)         X 
/iip2(l  -Pi)  +  h2pi{l  -  P2)  hip2{\  -  pi)  +  h2pi{l  -  P2) 

_  .    hiu-'^jhi  +  h2)p2{l  -pi)    X       .    h2U-'^{hi  +  /i2)pi(l  -P2)    x 

hiP2{l  -  Pi)  +  /l2Pl(l  -  P2)  hiP2{l  -  Pi)  +  /i2Pl(l  -  P2) 

ln(         '•^^  +^2)pi(l  -P2)         X  _  ,    h2U-^{hi  +  h2)pi{l  -P2)    X 

/llP2(l  -  Pi)  +  /l2Pl(l  -  P2)  ^1P2(1  -  Pi  )  +  h2pi{l  -  P2) 

+  h2i'-A.  (8.21) 
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The  In(-)  terms  cancel  out  and  upon  rearranging,  we  have 

V"(b*)  =  /  ^1^2(1  -  Pi)  +  ^2pi(l -P2)n/     /       hijhi  +h2)p2{l-  pi)       s 

I       h2(hi  +  h2)p2{l  —  P\)       \       ,         ,    \ 

-     A  ,    ',        ,,  J  +  /ii  +  /12    ,  8.22) 

hip2{l  -  Pi)  +  h2pi{l  -  P2)  ) 

which  equals  zero.     | 

We  now  extend  the  function  V*'  derived  in  Proposition  8.2  beyond  the  interval  [a*,  h*\ 
in  the  obvious  way  suggested  by  conditions  (6.12)-(6.13)  and  define  the  candidate  potential 

function  V*  in  terms  of  V*' .  Define  V*  by 

{  \^V'"{y)dy,         ifx>0, 
l/*(x)=      -'"     „  -  (8.23) 

\   -/;F*'(y)dy,     ifx<0, 

where 

{-r*,  if  X  <  a*, 

^*-  ifa:>6*,  (S.24) 

9L  +  C'e--^'  -  ^e-"-  /;.  h{y)t^ydy,  if  a*  <  x  <  h\ 

Notice  that  I'*  G  C^,  as  required,  and  that  F*(0)  =  0. 

The  candidate  solution  (g*,  F*(x),  r*,  I* ,a* ,  b*)  is  now  completely  specified,  and  equa- 
tions (6.10)-(6.15)  of  the  optimality  equations  have  been  verified.  To  complete  the  verifi- 
cation of  the  candidate  solution,  condition  (6.9)  needs  to  be  verified.  Since  (6.11)-(6.13) 
have  already  been  verified,  it  suffices  to  check  that 

TV*{x)  +  h{x)-g*  >0      for  x^  [a*,  6*]  (8.25) 

and 

-r*    <  T'*'(x)    <    r     for  a*  <  X  <  b* .  (8.26) 

Lemma  8.6.     Condition  (8.25)  holds. 

Proof.    To  show  (8.25),  notice  that  for  x  <  a*, 

TV*{x)  +  h{x)  —  g*  =  —pr*  —  h\X  —  g* 
>  —ur    —  hiQ    —  g 

=  0.  (8.27) 
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Similarly,  for  j  >  6*, 

TV*{x)  +  h{x)  -g*  =  fil*  +  h2X  -  g* 
>  ul*  +  h2b*  -g* 
=  0.  I  (8.28) 

Lemma  8.7.     Condition  (8.26)  holds. 

Proof.      Since  V*'{a*)  =  -r%V"{b*)  =  I'  and  V*"{a*)  =  V*"(b*)  =  0,  to  prove 
(8.26)  it  suffices  to  show  that 

V*"{x)  >    0     for  a*  <  X  <  b* .  (8.29) 

For  a*  <  X  <  0,  we  have,  by  (8.18)  and  integration  by  parts, 

V*"Ij:)  =      ^1  (         {hi  +h2)p2{l-  Pi)         .  ^,         (hi  +  h2)p2{l  -  Pi)         >.^ _^^ 
P     hip2il  -  p\)  +  h2piil  -  P2)  hip2{l  -  pi)  +  h2pi{l  -  P2) 

+  ^  -  ^^lil^(i.xe-  -  e-  -  j.G*e-'  +  e^').  (8.30) 

a-'  p,  ' 

Substituting  for  a*  from  (7.12)  and  cancelling  and  rearranging  terms  yield 

T-.//.    ^        ^1 /.  (/ii  + /?2)p2(l  -  ^1)  -^^^      f        *^      ^n  ro'J1\ 

V       (x)  =  — (I--; r: ^ ; re        )      for  a    <  a:  <  0  (8.31) 

p^         h,p2il-pi)  +  h2Pi{l-p2)  '  -  '         ' 

Letting  x  =  0  in  (8.31)  gives 

y '701  =  hih2ipi  -  P2) ,        . 

^   ^       p{hiP2{l-pi)  +  h2Pi{l-p2))'  ^   ■     ' 

which  is  greater  than  zero  by  definition  (4.2).  Also,  for  a  <  x  <  0, 

V*"'(x)  =  2hr{h^  +  h2)p2il-pi) 

^"^^       aHh,P2{l-Pi)  +  h2P,il-p2))'        '  ^^-^^^ 

which  is  also  greater  than  zero.    Since  V*"{a)  =  0,    F*"(0)   >  0  and  V*"'{x)  >  0  for 
X  G  (a*,  0),  it  follows  that  V*"{x)  >  0  for  a  <  x  <  0. 
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Similarly,  for  0  <  x  <  6*,  we  have  by  (8.18)  and  integration  by  parts, 

p^h,p2{l- Pi)  +  h2Pl{l-p2)'    ''^hiP2{l-pi)  +  h2Pi{l-p2)^^ 

2h2X       hie-"'  f  ^^  (hi  +  h2)p2{l  -  Pi) 


/i       V  hip2{l- pi)  + h2pi{l  -  P2) 

(ftl  +  /?2)P2(1  -Pl)  ,     /  [hi  +  h2)p2il  -  Pi) 

ln(- 


hiP2{l  -  pi)  +  h2pi{l  -  P2)         hip2il  -  Pl)  +  h2pi{l  -  P2, 
+  IHllZ  (e'^'ivx  -  1)  +  1  y  (8.34) 


which  can  be  rewritten  as 

.^^n,      .  h2  h2{hi   +  h2)pl{l   -  P2)  _^^         r         r.   ^         ^    U*  /C   OrA 

V      {x]  = + -— -— -e  forO<x<i.  (8.35 

P-        p{hip2{l- pi)  + h2pi{l  -  P2)) 

For  0  <  X  <  6*.  we  can  differentiate  V*"(x)  to  obtain 

.'*,>,,._  2h2{hi   +  h2)Pl{l  -  P2)  ,-^x  .oo^^ 

^    ^~'crHhiP,{l-pi)  +  h2Pi{l-p2))"        '  ^^-^^^ 

which  is  less  than  zero.  Since  V*"(0)  >  0,  V*"{b*)  =  0  and  V*"'{x)  <  0  for  x  G  (0,  6*),  we 
have  that  V*"{x)  >  0  for  0  <  x  <  6*.  Thus,  (8.29)  has  been  shown,  which  completes  the 
proof.     I 

We  are  now  in  position  to  state  the  solution  to  problem  (4.11)-(4.13). 

Theorem  8.8.  The  solution  to  problem  (4.11)-(4.13)  is  the  control  limit  policy 
{R',L*)  defined  in  (7.1)-(7.2)  on  the  interval  [a*,b*],  where  a*  and  b*  are  defined  in 
(7.12)-(7.13). 

Proof.  The  results  from  Lemma  7.1  through  Lemma  8.7  lead  to  a  solution  (g*, 
V'*(x),  r*, /*,  a*,  6*)  that  satisfies  equations  (6.9)-(6.15)  such  that  V  G  C^.  By  Theorem 
6.2,  the  result  holds  if  the  function  V*  satisfies  condition  (6.8)  that  V{x)  <  Ni  +  N2h{x) 
for  some  constants  A'^i  and  A^2-  This  condition  holds  by  setting 

Ni=m&x{V*{a*),V'{b*)}  (8.37) 
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and 


r*  +  /•    r*  +  I* 
N2  =  max{— 7 ,  —7 }.  I  (8.38) 

/l2  ^1 


As  is  usual  in  control  problems  dealing  with  long-run  average  cost  criterion,  the  op- 
timal poHcy  {R*,L*)  is  not  unique.  In  particular,  any  policy  that  brings  the  controlled 
Brownian  motion  process  within  the  interval  [a*,  6*]  in  a  finite  expected  amount  of  time 
and  then  uses  a*  and  b'  as  reflecting  barriers  will  achieve  the  same  average  expected  cost 
as  the  policy  {R*,L*). 

9.     The  Driftless  Case 

In  this  section,  the  constrained  problem  (4.11)-(4.13)  is  solved  vmder  the  conditions  of 
perfect  system  balance,  where  p\  =  P2-  In  this  case,  the  two-dimensional  Brownian  motion 
B  defined  in  (4.1)  has  drift  /x  =  0.  Defining  C,  by 

e  =  v^Pi(l-pi),  (9.1) 

it  is  seen  that  the  right  side  of  constraints  (4.12)  and  (4.13)  both  equal  1^.  Taking  ^  as 
an  essential  system  parameter,  problem  (4.11)-(4.13)  will  be  solved  in  terms  of  it.  The 
procedure  used  in  Sections  4  through  8  is  still  valid  in  this  case,  but  the  computations 
are  now  substantially  easier.  In  particular,  the  sufficient  conditions  for  optimality  are  the 
same  as  in  Theorem  6.2,  except  that  conditions  (6.14)-(6.15)  are  replaced  by 

limsnp^E,[R{t)]=\\msnp^E,[L{t)]  =  t  (9.2) 

As  in  Section  7,  the  first  step  towards  finding  a  solution  to  the  optimality  equations 
(6.9)-(6.13)  and  (9.2)  is  to  find  candidate  interval  endpoints  a*  and  b* .  See  Chapter  5  of 
Harrison  [4]  for  a  proof  of  the  following  result,  which  is  the  analogue  of  Lemma  7.1  for  the 
driftless  case. 
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Lemma  9.1.  Suppose  B  is  a  (0,(7^)  Brownian  motion,  R  and  L  are  as  defined  in 
(6.1)-(6.2),  and  thus  W  =  B  +  R  —  L  is  a  regulated  Brownian  n^otion  on  the  interval  [a,  b]. 
Then  W  has  a  uniform  steady  state  distribution  on  [a,  b]  and 

limsup  ^E4R{t)]  =  limsup  ^Er[L(t)]  =      ^  (9.3) 

T—'OO     ■'■  T—<-oo     ■'■  ^\0        (^) 


The  candidate  interval  endpoints  are  derived  by  solving  the  following  problem:  sujiong 
the  class  of  control  limit  policies,  find  the  policy  {R,  L)  to 

•  r 


minimize    iim 


limsup  i£,[/     h{W{t))dt]  (9.4) 

T^oo      J  Jo 


subject  to    limsup  —E^[R[t)]  =  limsup  —E^[L{t)]  =  (9.5) 


Lemma  9.2.     The  control  limit  policy  with  interval  endpoints 


* 

h2       a'^ 

h,  +h2  2C 

b*  = 

h,       a^ 

(9.6) 


(9.7) 


and 

b*  = 

hi  +  h2  2e 

solves  problem  (9.4)-(9.5). 

Proof.    Equating  the  right  sides  of  (9.2)  and  (9.5)  leads  to  the  requirement 

2 

6-a=^.  (9.8) 

Thus  a*  can  be  found  by  minimizing  /(a)  over  the  interval 

-^   <   a   <   0,  (9.9) 

where 

Of     r°  /•«  +  ff 

f(a}^-^{        -hixdx+  h2xdx).  (9.10) 
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Integrating  gives 


i         <' 


Setting  /'(a)  =  0.  we  have 


J         2 

2/iia  +  2/i2a+  ^—  =  0 


(9.11) 


(9.12) 


Solving  for  a  yields  a*  in  (9.6)  and  using  (9.8)  to  solve  for  6  yields  b*  in  (9.7).  These  are 
the  cajididate  interval  endpoints,  since 


/"(a)  =  4(2/^1 +2/12), 

which  is  positive.     | 

Since,  upon  cancelling  and  rearrcinging  terms, 


(9.13) 


/(a*)  = 


a'^h^h: 


the  candidate  gain  g*  is 


4^/11  +  /l2)' 
(T^hih2 


(9.14) 


+  ri  +  li. 


Theorem  9.3.    Let  X*  e  [0, 1]  and  let  a*  and  b*  be  deRned  as  in  (9.6)-(9.T).  Let 


(9.15) 


'*  =  A* 


h,  +  h2Ae' 


r  =  (i-y)  ^''^    "' 


hi  +  h2  4^2  ' 

hih2     cr^ 


hi  +h2  2C 


and 

V'ix)  = 
where 

V*'{x)  =  { 
and 


J^'V*'iy)dy,  ifx>0, 

-j'V*'{y)dy,     ifx<0, 

'  — r*,  if  X  <  a*, 

/*,  ifx>b*, 

'-^  -  ^  i:.  hiy)dy  +  C; ,     ifa'<x<b\ 


(9.16) 
(9.17) 
(9.18) 

(9.19) 
(9.20) 


c; 


,     (7'(-yhih2  +  i2-y)hihi) 


HHh,+h2)^ 


(9.21) 
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Then  {g%  V*{x).r\l\a\b*)  solves  equations  (6.9)-(6. 13)  and  (9.2).  Furthermore,  V  e  C^ 
and  satisfies  condition  (6.8). 

Thus,  by  Theorem  6.2,  the  solution  to  problem  (4.11)-(4.13)  in  the  driftless  case  is  the 
control  limit  pohcy  (i?*,  I")  defined  by  (7.1)-(7.2)  on  the  interval  [a*,  6*],  where  a*  and  b* 
are  defined  in  (9.6)-(9.7).  The  proof  of  Theorem  9.3  parallels  the  arguments  in  Section  8 
that  lead  to  Theorem  8.8.  However,  the  computations  in  this  case  are  substantially  simpler 
and  will  be  omitted.  This  section  is  concluded  by  showing  that,  as  the  drift  ;/  — >  0,  the 
endpoints  derived  in  Section  8  for  the  nonzero  drift  case  converge  to  the  endpoints  derived 
in  the  driftless  case. 

Proposition  9.4. 

lim'^'lnf         (^1+^2)P2(1-Pi)         \_  h       a^ 

and 

<y\     f         {hi  +  h2)pi{l-p2)         \  hi       a^ 


lim  ^In '    '   '     "^'y      '^'' =  —li—  —  .  (9.23) 

M-o2^       \hip2{\-pi)  +  h2Pi{l-p2)J       hi  +  h2  2i  ^        ' 


Proof.    Since  p  =  \/n(pi  —  P2).  it  follows  that 

,.      <^\     f         {hi  +  h2)p2{l  -  Pi) 
iim  - —  in 


/i- 


-0  2/j       \hip2{l  -  Pi)  + h2pi{l- P2] 


lim    -yj^ In  f        ih,+h}Ml-p,)  ,  , 

P2-P1  2y/n(pi  -  P2)       \hip2{l  -/9i)  +  /i2Pi(l  -  P2) 


By  THopital's  rule,  one  obtains 


,.                 ^^  f         (hi+h2)p2(l-pi) 

hm    - — ■=r, In 


P2-PI  2-/n{pi  -  P2)       \hip2{l  -  pi)  +  h2Pi{l  -  p2, 


J_(y2  Jj^  /         (/ii+/i;)P2(l-Pi)  ^ 

(Jp2  \  ''lP2(l-Pl)  +  'l2Pl(l-P2)  / 


=    lim    — \   "^^^   ^'^      ^^"     •^''^  (9.25) 

^    ,.        -a'^pih2 

P2-PI  2yynp2(hip2{l  -  pi)  +  h2pi{l  -  P2)) 


-a^h 


2 


2y/^{hl    +    h2)pi{l    -   PlY 
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(9.27) 


The  definition  of  ,^  in  (9.1)  completes  the  proof  for  the  endpoint  a* .    The  proof  for  b*  is 
similax  and  is  omitted.     | 

10.     Solution  to  the  Workload  Formulation:  Summary 

The  purpose  of  this  paper  was  to  obtain  a  solution  (U,Z,d)  to  the  workload  formu- 
lation (2.6)-(2.11)  of  the  limiting  control  problem.  This  is  a  self-contained  section  that 
summarizes  the  solution.  The  parameters  p,,  M,jt,  t'l,  cr^,  h,,  u  and  ^  appearing  in  this  sec- 
tion are  all  defined  in  terms  of  the  primitive  problem  data.  Their  definitions  can  be  found 
in  equations  (l.S).  (2.1),  (2.3),  (4.3),  (4.7)-(4.S),  (7.3)  and  (9.1),  respectively. 

In  the  workload  formulation,  the  controller  observes  a  two-dimensional  Brownian  mo- 
tion process  B,  from  which  can  be  observed  the  one-dimensional  Brownian  motion  process 
B  defined  by 

B{t)  =  p.Biit)  -  p,B2{t),  t>0.  (10.1) 

If  pi  ^  P2,  then  define  the  interval  endpoints  a  and  b  by 

-1,       f  {h,+h2)p2{l-p,)  \  ,.„_. 

a  =  1^       In -; ^ ; -; (10.2) 

\hiP2il-pi)  +  h2Pi{l-p2)J  ^         ' 


and 


If  p\  =  P2>  then  let 


anc 


t  =  ,-.  In  (  — l^i±MMilL£lL^  I  .  (10.3) 

hip2{l  -  P\)  +  h2pi{l  -  P2), 


h2       a'' 
hi+h2  2^ 


(10.4) 


For  a  particular  realization  of  B,  define  the  control  functionals  (R,  L)  by 

R{t)=    sup  [a-B(5)  +  L(5)]+  (10.6) 

0<s<J 
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and 


L[t)  =    snp  [B{s)  +  R{s)  -  b]^ .  (10.7) 

0<s<t 


The  two-dimensional  optimal  control  process  U  is  given  by 


and 


U^{t)  =  ^  (lO.S) 

P2 


IW)  =  ^.  (10.9) 

Pi 


From  the  functionals  {R,L)  in  (10.6)-(10.7),  next  define  the  process  W  by 

W{t)  =  B(t)  +  R{t)  -  L{t)     foralH>0.  (10.10) 


The  7\.'-dimensional  optimal  control  process  Z  is  given  by 

— 57 —T! — ,     if  A;  =  1  an( 

0,  if  ^  7^  1  and  Wit)  >  0. 


,,^'^^\,    ,     if  it  =  1  and  W{t)  >  0, 
Zi-(f)=  <(   ''^^^"~^^^^^'  .  (10. ii: 


and 


vv«) 


-,     if  A:  =  2  and  Wit)  <  0, 


Zkit)={   P2W12-P1A/22'  -  '  (10.12) 

[0,  if  fc  7^  2  and  W{t)  <  0. 

Finally,  the  optimal  control  process  0  is  given  by 

Oit)^v-^[Biit)  +  —^-y^AhkZkit)],    foralU>0.  (10.13) 

Thus  the  solution  iU,Z,9)  to  the  workload  formulation  (2.6)-(2.11)  is  given  by  equations 
(10.S)-(10.9)  and  (10.11)-(10.13). 
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